-
Notifications
You must be signed in to change notification settings - Fork 640
Refactor pagination to prepare for cursor-based pagination. #1807
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
This continues the changes made in rust-lang#1763. Since that PR was merged, one of the non-code steps has been taken care of -- All users hitting any endpoint with `?page=20` (which is an arbitrary search pattern that seemed high enough to give any crawlers going through pagination) have been contacted about the change, with a PR opened against any that included a repo. (Intersting aside, there are *zero* records of this for any endpoint other than search, which perhaps implies we can get rid of a few of these endpoints, but that's a separate discussion). This PR does not change any functionality, but moves some code around to better encapsulate things for upcoming changes. Specifically: - Change our frontend to show "next/prev page" links on the all crates page - Stop returning the "total" meta item when the next/prev page links will be cursor based (which I'd actually just like to start omitting in general) The main goal of this change was to stop having any code outside of `Paginated` (which has been renamed to `PaginatedQuery`, as there's a new type called `Paginated`) care about how pagination occurs. This means other code can't care about *how* pagination happens (with the exception of `reverse_dependencies`, which uses raw SQL, and sorta has to... That was a bit of a rabbit hole, see diesel-rs/diesel#2150 for details. Given the low traffic to that endpoint, I think we can safely ignore it). The distribution of responsibilities is as follows: - `PaginatedQuery<T>`: Given the query params, decides how to paginate things, generates appropriate SQL, loads a `Paginated<T>`. - `Paginated<T>`: Handles providing an iterator to the records, getting the total count (to be removed in the near future), providing the next/prev page params - `Request`: Takes the pagination related query params, turns that into an actual URL (note: Due to jankiness in our router, this will only ever be a query string, we have no way of getting the actual path) The next step from here is to change our UI to stop showing page numbers, and then remove the `total` field. This PR will introduce a performance regression that was addressed by rust-lang#1668. That PR was addressing "this will become a problem in a future", not "this is about to take the site down". Given the intent to remove the `total` field entirely, I think it is fine to cause this regression in the short term. If we aren't confident that the changes to remove this field will land in the near future, or want to be conservative about this, I can add some additional complexity/git churn to retain the previous performance characteristics
r? @jtgeibel (rust_highfive has picked a reviewer for you, use r? to override) |
It seems clippy is still not happy. https://travis-ci.com/rust-lang/crates.io/jobs/226409422#L499 |
I swear I run this locally...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This PR looks great to me, and I saw no issues when testing locally.
Feel free to include this in the next deploy if you want. Otherwise I'll r+ soon.
@bors r=jtgeibel |
📌 Commit 8ef5179 has been approved by |
Refactor pagination to prepare for cursor-based pagination. This continues the changes made in #1763. Since that PR was merged, one of the non-code steps has been taken care of -- All users hitting any endpoint with `?page=20` (which is an arbitrary search pattern that seemed high enough to give any crawlers going through pagination) have been contacted about the change, with a PR opened against any that included a repo. (Intersting aside, there are *zero* records of this for any endpoint other than search, which perhaps implies we can get rid of a few of these endpoints, but that's a separate discussion). This PR does not change any functionality, but moves some code around to better encapsulate things for upcoming changes. Specifically: - Change our frontend to show "next/prev page" links on the all crates page - Stop returning the "total" meta item when the next/prev page links will be cursor based (which I'd actually just like to start omitting in general) The main goal of this change was to stop having any code outside of `Paginated` (which has been renamed to `PaginatedQuery`, as there's a new type called `Paginated`) care about how pagination occurs. This means other code can't care about *how* pagination happens (with the exception of `reverse_dependencies`, which uses raw SQL, and sorta has to... That was a bit of a rabbit hole, see diesel-rs/diesel#2150 for details. Given the low traffic to that endpoint, I think we can safely ignore it). The distribution of responsibilities is as follows: - `PaginatedQuery<T>`: Given the query params, decides how to paginate things, generates appropriate SQL, loads a `Paginated<T>`. - `Paginated<T>`: Handles providing an iterator to the records, getting the total count (to be removed in the near future), providing the next/prev page params - `Request`: Takes the pagination related query params, turns that into an actual URL (note: Due to jankiness in our router, this will only ever be a query string, we have no way of getting the actual path) The next step from here is to change our UI to stop showing page numbers, and then remove the `total` field. This PR will introduce a performance regression that was addressed by #1668. That PR was addressing "this will become a problem in a future", not "this is about to take the site down". Given the intent to remove the `total` field entirely, I think it is fine to cause this regression in the short term. If we aren't confident that the changes to remove this field will land in the near future, or want to be conservative about this, I can add some additional complexity/git churn to retain the previous performance characteristics
☀️ Test successful - checks-travis |
This continues the changes made in #1763. Since that PR was merged, one
of the non-code steps has been taken care of -- All users hitting any
endpoint with
?page=20
(which is an arbitrary search pattern thatseemed high enough to give any crawlers going through pagination) have
been contacted about the change, with a PR opened against any that
included a repo. (Intersting aside, there are zero records of this for
any endpoint other than search, which perhaps implies we can get rid of
a few of these endpoints, but that's a separate discussion).
This PR does not change any functionality, but moves some code around
to better encapsulate things for upcoming changes. Specifically:
page
will be cursor based (which I'd actually just like to start omitting
in general)
The main goal of this change was to stop having any code outside of
Paginated
(which has been renamed toPaginatedQuery
, as there's anew type called
Paginated
) care about how pagination occurs. Thismeans other code can't care about how pagination happens (with the
exception of
reverse_dependencies
, which uses raw SQL, and sorta hasto... That was a bit of a rabbit hole, see
diesel-rs/diesel#2150 for details. Given the
low traffic to that endpoint, I think we can safely ignore it).
The distribution of responsibilities is as follows:
PaginatedQuery<T>
: Given the query params, decides how to paginatethings, generates appropriate SQL, loads a
Paginated<T>
.Paginated<T>
: Handles providing an iterator to the records, gettingthe total count (to be removed in the near future), providing the
next/prev page params
Request
: Takes the pagination related query params, turns that intoan actual URL (note: Due to jankiness in our router, this will only
ever be a query string, we have no way of getting the actual path)
The next step from here is to change our UI to stop showing page
numbers, and then remove the
total
field.This PR will introduce a performance regression that was addressed by
#1668. That PR was addressing "this will become a problem in a future",
not "this is about to take the site down". Given the intent to remove
the
total
field entirely, I think it is fine to cause this regressionin the short term. If we aren't confident that the changes to remove
this field will land in the near future, or want to be conservative
about this, I can add some additional complexity/git churn to retain the
previous performance characteristics